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ABSTRACT 

The standards developed for reform in mathematics 
education by the National Council of Teachers of Mathematics (NCTM) 
de - emphasize the algorithmic manipulation of mathematical symbols. 

The NCTM takes the position that the rapidly changing workplace is an 
environment that requires problem-solving skills with computers being 
used to perform routine mathematical computations. This report 
describes the computer-based information curriculum module developed 
for beginning algebra to meet the challenges of the proposed reform. 
The RAND Algebra Tutor Project has been developing and piloting 
intelligent computer tools for first-year algebra students. Varios 
problems led to the development of a full curriculum in order to 
successfully implement computer-assisted instruction. The curriculum 
departed from traditional algebra instructions in these ways: the 
pedagogy stressed situational learning in real-life problems, 
introducing a conceptual framework before local skills practice and 
used indictive instruction and cooperative learning. The curriculum 
achieved mixed effectiveness. In general, students were more 
successful at creating and using qualitative models than quantitative 
models. Appendix A contains the 12 teacher lessons plans for the 
linear equations unit and related student materials. Appendix B 
contains the evaluation measures for the arithmetic text, pre/post 
tests, student course evaluations, and unit evaluations. (Contains 36 
references.) (MDH) 



it * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Vc * * * * * * * * * * * * * x * x * * * * * 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 



0&3 U*)/ 



W 



A RAND NOTE 



Implementing a Novel Computer-Related 
Algebra Course 



Abby Robyn, Cathleen Stasz, 
David McArthur, Tor Ormseth, 
Matthew Lewis 



BEST COPY AVAILABLE 



RAND 

^er|c 

v; 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Elizabeth Gill 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC).” 



U S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 



This document has been reproduced as 
I-! received from the person or organization 
originating it 

□ Minor changes have been made to improve 
reproduction quality. 

• Points of view or opinions stated m this docu- 
ment do not necessarily represent official 
OERI position or policy. 



The research described in this report was supported by the National Science Foundation 
under Grant No. R168D90023 and by RAND using its own research funds. 



RAND is a nonprofit institution that seeks to improve public policy through 
research and analysis. Publications of RAND do not necessarily reflect the 
opinions or policies of the sponsors of RAND research. 



o 

C) 



Published 1992 by RAND 

1700 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 



A RAND NOTE 



N-3326-NSF/RC 



Implementing a Novel Computer-Related 
Algebra Course 



Abby Robyn, Cathleen Stasz, 
David McArthur, Tor Ormseth, 
Matthew Lewis 



Supported by the 
National Science Foundation 



4 



RAND 



- iii - 



PREFACE 



The research reported in this Note is part of an ongoing series of studies to develop 
and field test an intelligent computer tutor for basic algebra and describes, in particular, a 
model for transferring technology into the classroom. This project was supported by RAND, 
using its own funds; and it incorporated software developed with a grant from the National 
Science Foundation as a part of its Applications of Advanced Technologies Program. 
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SUMMARY 



The crisis in education articulated in A Nation At Risk (1983) and the Carnegie report 
(1986) has focused extensively on the poor mathematics performance of American students. 
This concern is reflected in the 1986 National Assessment of Education Progress (Dossey et 
al., 1988) and in the complaints of employers that workers do not have appropriate skills to 
contribute successfully to a technology-dominated workplace (Romberg, 1987). In response to 
the call for reform, the National Council of Teachers of Mathematics (NCTM) proposed a new 
set of standards for K-12 mathematics instruction (Romberg, 1987). It suggests a variety of 
fundamental content changes for mathematics education. At the most general level, the 
NCTM standards emphasize the need for students to learn higher-order thinking and 
problem-solving skills. Conversely, the standards deemphasize the rote or algorithmic 
manipulation of mathematical symbols, arguing both that the rapidly changing workplace 
demands the acquisition of flexible problem-solving skills and that computers should take 
over the computation of routine mathematical results. 

The NCTM standards lead to several specific questions concerning the content of 
mathematics courses and the role of computers in the classroom. What problem-solving and 
higher-order thinking skills should replace traditional ones in the classroom? What 
techniques should be used to teach these new skills? How should computers be introduced 
into these courses to support students’ thinking by doing routine mathematical 
computations? Should computers be used simply as a problem-solving resource or should 
they take a more active role in teaching students? 

1116 RAND Algebra Tutor Project has been developing and piloting intelligent 
computer tools for first-year algebra students. However, inadequate training, new 
instructional, pedagogical, and organizational goals, and logistical problems all contribute to 
the difficulty of effectively implementing computer-assisted instruction. It has become clear 
to us that successful implementation of a technology-based course requires the development 
of a full curriculum, comprising a set of support materials that permit the classroom culture 
to change to take advantage of the tools and new curriculum content. In this Note, we 
describe the curriculum module for beginning algebra that we developed to meet the 
challenges discussed above. 
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THE CURRICULUM 

The curriculum we developed departed from traditional algebra instruction in three 
ways: The pedagogy stressed new techniques situating learning in real-life problems, 
introduced a conceptual framework before local skills practice, and used inductive instruction 
and cooperative learning; the course content included innovative topics -modeling, covariate 
relationships, and problem-solving strategies; and the teaching tools included daily access to 
an intelligent tutoring system. A multidisciplinary team composed of computer scientists, 
cognitive and educational psychologists, implementation and curriculum experts, and 
teachers developed the cognitive goals for the course and the course materials. Curriculum 
materials included teacher lesson plans, parallel computer and non-computer activities for 
students, and evaluation instruments. 

Content Goals 

The course subject matter centered on teaching students various model-building skills, 
similar to those noted in NCTM and other related research (e.g., Fey, 1984; Thompson and 
Rathmell, 1988; Usiskin, 1985). By model-building skills we mean the ability to see how 
formal mathematical objects can relate to the real world — for example, how equations can 
provide powerful descriptions, or models, of the interactions among real-world objects or 
properties. Model building includes several skills rarely taught in traditional classrooms: 
formulating which objects or properties of a real-world situation should be represented as 
mathematical objects or variables; data gathering, representation, and building qualitative 
models; translating qualitative models into quantitative ones; and making inferences about 
the behavior of variables. 

Pedagogical Goals 

The pedagogical goals adhered to the principles that problem solving should reflect the 
real world and take advantage of computer capabilities. Thus, coursework problems should 
be grounded in natural situations such as after school jobs, rock concerts, and pollution, and 
problem solving should reflect the cooperative arrangements frequently found in the 
workplace. Moreover, whenever possible, learning should occur through induction rather 
than didactic presentation, and global skills such as strategic goals for problem solving 
should be taught before the skills of symbol manipulation. 

Computer Tools 

Two computer environments were built to help students engage in each of the main 
activities involved in model building, formulation, data gathering and representation, 
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translation, and inferendng using equations. Curriculum software was implemented in 
Franz Lisp on Sun Microsystems workstations (see McArthur et al. 1987; McArthur and 
Stasz, 1990, for a more complete description of the tutor architecture and functionality). For 
each assignment, students select a situation from a menu of options (e.g., video store job, 
pizza party, weight estimation, boating). Students determine the variables in a situation and 
insert them into a word graph which forms a qualitative model of the variable relationships. 
Besides the initial situation and word graph, the model-building computer environment 
provides several tools to help the student gather and organize data, including tables of values 
and cartesian graphs. Problems posed by the computer are aimed at helping students 
develop an intuitive or qualitative model of situations which in later sessions will be 
translated into a more quantitative, equational representation. 

In later computer sessions, additional tools appear to help students construct 
operators and then symbolic equations. The students then use their equational models to 
generate answers to problems about the topic they have chosen. It is at this point that an 
equation- solving intelligent tutor system (ITS) comes into play, coaching the student through 
the manipulations required to solve equations and make the desired inference. Specific 
assignments are structured to focus on what goal to pursue in solving an equation (e.g., 
isolating a variable from its coefficient), what operation implements the goal (e.g., dividing 
both sides of the equation by 5), and how to manipulate symbols to accomplish an operation. 
Consistent with our top-down pedagogical philosophy, in equation-solving assignments, 
students first focus only on selecting appropriate goals, and the ITS supplies operations and 
manipulations. Later, students become responsible for operations, and in the final 

assignments students do goals and manipulations. 

0 

Teacher and Student Materials 

To provide support for teachers and insure that the course material would be taught as 
intended, we developed specific lesson plans (see Appendix A for the full set of curriculum 
materials). Each lesson plan contains the goals for the lesson and an almost scripted set of 
activities to carry out, including board examples for the teacher to use. In addition, eight 
computer and 12 noncomputer assignment worksheets were developed. 

IMPLEMENTATION 

The curricul um module was taught by two teachers who were consultants to the 
curricul um development team. Participating students were enrolled in either Math A or 
regular algebra classes. Math A is the first year of a new two-year course implementing the 
recent California Math Framework (California State Department of Education, 1985). It 
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features several discrete units — probability, symmetry, statistics, algebra, and geometry. 

The Math A course is intended to expose low-achieving mathematics students to enough 
algebra and geometry to enable them to continue in a college-focused academic plan if they so 
choose. To provide comparative information, four regular algebra classes were asked to 
participate in the data-collection activities. 

Training 

Since both pilot teachers were members of the curriculum development team, formal 
teacher training was required only for the computer component of the course. The software 
was available for teacher review as it was developed. Additionally, some assignments were 
piloted with students in a lab situation. Seven Sun 3/50 workstations were installed in the 
Math A classroom. Project staff were present in every class to handle any computer 
operational problems that arose. 

Assessment 

We used multiple methods to assess the curriculum, including mathematics and 
algebra achievement tests, two unit tests, background and attitude questionnaires, classroom 
observation notes, and computer records (see Appendix B for the assessment instruments). 

EVALUATION AND CONCLUSIONS 

This study had two broad goals: testing a novel approach to algebra that focused on 
helping students learn skills for building and using mathematical models to solve real 
everyday problems, and developing techniques for the successful transfer of technology into 
the classroom. Overall, the technology transfer was effective, insofar as the computers 
functioned as a standard component of daily lesson activities, typically used by each student 
twice a week. Thus, the technology transfer model, whereby a team of researchers and 
teachers provide computer-integrated lesson plans and course materials, proved successful in 
enabling substantial computer use for new activities. However, teacher comfort with the 
technology varied greatly between the two teachers. Increased training that includes 
sufficient time for teachers to become familiar with computer capabilities and assignments 
and the operation of the machines themselves is necessary to help teachers become more 
comfortable with the new teaching tool. 

The novel algebra curriculum was intended to teach students to acquire several skills, 
including identifying the variables in a real-world situation; describing the covariation of the 
values of dependent variables; recognizing and interpreting multiple representations of 
information (tabular, graphical, and equational); constructing qualitative and quantitative 
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models of situations; solving problems on several levels (specifying solution goals and 
operations and performing symbol manipulations); and interpreting solutions in terms of the 
original problem. The curriculum achieved mixed effectiveness. In general, students were 
more successful at creating and using qualitative models than quantitative models. In the 
qualitative modeling section of the curriculum, over two-thirds of the students could 
successfully identify the variables, build qualitative models, gather data to explore 
relationships, interpret tabular and graphical representation, and describe relationships. On 
these innovative course items, they outperformed the algebra students, a significant result 
given that the Math A students’ basic math abilities were significantly less than their peers 
in the algebra classes. 

Students were less successful at constructing quantitative models and equation 
solving. The class unit tests indicate that only about a quarter of the students could specify 
the three operators necessary to transform a word graph to a word equation; about half of the 
students could specify the appropriate goals for equation-solving steps; about a third could 
specify the appropriate operation; 45 percent could perform the symbol manipulations to 
solve problems arising from real-world situations; and 24 percent could perform the symbol 
manipulations to solve abstract problems. 

Four factors appear to have influenced our results: (1) The target population we 
selected had limited mathematical skills. Though final outcomes were low, we were 
encouraged by the substantial gains in skills made by students during the study. (2) The 
teachers had difficulty in adopting a new approach to algebra and new roles in the classroom 
despite extensive involvement in the course development. Our class observations often noted 
the teachers’ difficulties in assimilating new theoretical goals. For example, the teachers 
tended to teach the goals of particular steps in problem solving as processes, emphasizing 
what to do, not why to do it. The curriculum also introduced challenging roles for the 
teachers including facilitating student computer activities and acting as computer trouble 
shooter. The teachers displayed varying levels of comfort with computer activities indicating 
again the importance of training experiences to acquaint teachers with innovative 
approaches to mathematics and new course tools and materials. (3) A third factor 
contributing to our results was the slowness of the computers and frequent bugs causing 
computer crashes. Students’ time at the computers was restricted and the computers did not 
entirely assume the teaching role they were expected to have. (4) Finally, to understand our 
results, we looked more closely at the equation-solving activities in the curriculum, which 
were the activities that students found most difficult. We had expected that the familiarity 
with problem-solving goals and operations would produce an easy transition to direct symbol 
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manipulation. Students had only one computer assignment to practice direct symbol 
manipulation. Students’ poor equation-solving skills may indicate that more practice in 
direct symbol manipulation is desirable or that the top-down approach to problem solving 
was not effective in promoting acquisition of equation-solving skills. Students may require 
more familiarity with symbol manipulation before they are able to specify problem-solving 
goals. Though their overall posttest equation-solving scores were low, students did make 
significant gains in their skills, particularly in nonabstract equation solving. Students were 
twice as successful in solving equations that arose from real-world situations as they were in 
solving exactly the same type of equation presented abstractly. We speculate that situating 
equations allow students to draw on their intuitive and experiential resources and is more 
intrinsically motivating. 

NEXT STEPS 

Our next step is an expanded version of the curriculum module described above to 
include material on statistics. We will use the technology transfer model successfully fielded 
in this pilot but will devote more attention to teacher training and teacher assimilation of 
course goals. 
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1. INTRODUCTION 



The crisis in education articulated in A Nation At Risk (1983) and the Carnegie report 
(1986) has focused extensively on the poor mathematics performance of American students. 
This concern is reflected in the 1986 National Assessment of Educational Progress (Dossey et 
al., 1988) and in the complaints of employers that workers do not have appropriate skills to 
contribute successfully to a technology-dominated workplace (Romberg, 1987). In response to 
the call for reform, the national council of teachers of mathematics (NCTM) proposed a new 
set of standards for K-12 mathematics instruction (Romberg, 1987). It suggests a variety of 
fundamental content changes for mathematics education. At the most general level, the 
NCTM standards emphasize the need for students to learn higher-order thinking and 
problem-solving skills. Conversely, the standards deemphasize the rote or algorithmic 
manipulation of mathematical symbols, arguing both that the rapidly changing workplace 
demands the acquisition of flexible problem-solving skills and that computers should take 
over the computation of routine mathematical results. 

The NCTM standards lead to several specific questions concerning the content of 
mathematics courses and the role of computers in the classroom. What problem-solving and 
higher-order thinking skills should replace traditional ones in the classroom? What 
techniques should be used to teach these new skills? How should computers be introduced 
into these courses to support students’ thinking by doing routine mathematical 
computations? Should computers be used simply as a problem-solving resource or should 
they take a more active role in teaching students? 

The RAND Algebra Tutor Project has been working for the last four years on 
developing and piloting intelligent computer tools for first-year algebra students. Our work 
began with the development of an intelligent tutoring system (ITS) to help students learn 
symbol manipulation skills for solving problems in basic algebra (McArthur, 1987; McArthur 
et al., 1987; McArthur et al., 1988; McArthur and Stasz, 1990). Consistent with the NCTM 
standards, subsequent versions of our ITS have focused on helping students learn more 
“strategic” reasoning skills in algebra, leaving the “tactics” of symbol manipulation to the 
computer. For example, one version decomposes solving equations into several levels of 
decisionmaking and requires students to specify only the goals or operations needed to solve 
an equation (e.g., “collect terms” or “add 5 to both sides”) whereas the ITS executes the lower- 
level symbol manipulation that implements these instructions. 
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We have found that in partnerships where much of the underlying mathematical 
manipulation is being handled by the computer, students have opportunities to focus on 
important aspects of problem solving (e.g., defining variables, describing relationships among 
variables, developing strategies for problem solution) that are often difficult to include in 
traditional classroom instruction. However, we have also found that shifting the emphasis of 
our algebra tutor from more conventional content and pedagogy to such partnerships leads to 
problems integrating the technology into the classroom. Like other tutoring systems 
(Schoenfeld and Verban, 1988, Strudler and Gall, 1988), the algebra tutor suffered from 
implementation difficulties (see Robyn et al., 1989; Stasz et al., 1989). 

Inadequate training, new instructional, pedagogical, and organizational goals, and 
logistical problems all contributed to the difficulty of effectively implementing computer- 
assisted instruction. We found that the classroom culture placed priority on covering a given 
set of topics in a certain amount of time. Topics were taught didactically, and the pace of the 
class was fast and textbook driven. The teacher and students were intent on covering 
material relevant to the textbook and traditional test items. Tests emphasized skill in 
abstract symbol manipulation. In addition, the teacher did not monitor students’ work at the 
computer, limiting her coaching to students engaged in noncomputer activity. As a result, 
the nontraditional computer approach that we introduced, emphasizing higher-level 
cognitive skills of problem-solving strategy, was not evaluated as part of the coursework and 
hence had little value for students or teachers. 

Given these fairly typical classroom features (Tobin, 1987), it has become clear that 
implementing a technology-based course requires more than just the development of 
computer tools that focus on novel mathematical skills. Successful implementation demands 
the development of a full curriculum, comprising a set of support materials that permit the 
classroom culture to change to take advantage of the tools and new curriculum content. 
Supports include lessons that directly integrate computer activities into the coursework, 
training for teachers in managing the curriculum and technology, and a class setting freed 
from having to teach to traditional tests and from traditional instructional sequences. We 
believe these materials are as important as content and technology changes in improving the 
quality of learning of mathematics in our classrooms. 

In this Note we describe the curriculum module for beginning algebra that we 
developed to meet the challenges discussed above. Our curriculum by no means addresses all 
the problems associated with the reform of mathematics education and with the use of 
computers in the classroom. Rather, it should be viewed as a prototype computer-based 
course that attempts to address, in an integrated rather than piecemeal way, some problems 
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of mathematics curriculum content reform, the novel use of computers in the classroom, and 
implementation problems that arise from these content and technology changes. 

In Section 2 we describe the development and content of the curriculum. We discuss 
our overall approach to curriculum development, the specific novel mathematical topics we 
targeted, as well as the design of the computer software and teacher and student materials. 
Section 3 discusses the implementation of the module, student and teacher training 
procedures, and our assessment procedures. Section 4 evaluates the curriculum and its 
implementation, and we conclude in Section 5 with a discussion of the study results and 
proposed modifications of the curriculum. 



-4- 



2. THE CURRICULUM 



The curriculum we developed departed from traditional algebra instruction in three 
ways: The pedagogy stressed new techniques situating learning in real-life problems, 
introduced a conceptual framework before local skills practice, and used inductive instruction 
and cooperative learning; the course content included innovative topics — modeling, covariate 
relationships, and problem-solving strategies; and the teaching tools included daily access to 
an intelligent tutoring system. Curriculum materials included teacher lesson plans, parallel 
computer and noncomputer activities for students, and evaluation instruments. In this 
section we elaborate on these aspects of the curriculum beginning with our overall approach 
to curriculum development. 

APPROACH TO CURRICULUM DEVELOPMENT 

Designing a new curriculum that encompasses the features outlined above is a 
complex undertaking that requires different kinds of expertise. Although computer scientists 
and programmers can design computer-based learning tools, they may have little knowledge 
of how to use computers in classrooms. To ensure the viability of computer tools and, more 
generally, to develop a curriculum that would realistically transfer from the context of 
development to the context of actual use, we formed a multidisciplinary curriculum 
development team. This team — composed of computer scientists, cognitive and educational 
psychologists, and implementation and curriculum experts — determined the cognitive goals 
for the course and the basic course outline. The proposed outline was then reviewed by 
several teachers, two of whom joined the development team. Two subgroups formed to 
develop the course package; one group concentrated on computer software and the other on 
teacher lesson plans. Throughout the three-month development period both teams met 
frequently to ensure that software design meshed with other curriculum activities. The 
highly interactive process led to frequent adjustments and modifications as materials were 
actually developed and tested with a few students in a laboratory setting. 

Figure 1 illustrates how we conceived the design task. Central to our planning were 
content goals — the specific kinds of knowledge and skills we wanted students to learn — and 
pedagogical goals — the ways of learning and teaching course material we wished to promote. 
These informed the development of software tools and lesson plans and dictated the 
evaluation measures. Underlying this framework was a consistent concern that the end 
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Figure 1 — Design Tasks . 

result be viable in the classroom. Thus, we also considered such issues as the classroom 
culture and teacher and student comfort with the curriculum. 



CONTENT GOALS 

The course subject matter centered on teaching students various model-building skills, 
similar to those noted in NCTM and other related research (e.g., Fey et al., 1984; Thompson 
and Rathmell, 1988; Usiskin, 1985). By model-building skills we mean the ability to see how 
formal mathematical objects can relate to the real world, for example, how equations can 
provide powerful descriptions, or models, of the interactions among real-world objects or 
properties. Model building includes several skills rarely taught in traditional classrooms: 



Formulation. In a given real-world situation, formulation involves deter minin g 
which objects or properties should be represented as mathematical objects or 
variables. For example, in modeling the finances of a car wash to raise money to 
send the marching band to a distant competition, one might decide that 
properties such as the number of cars and the amount one has to spend on 
supplies should be variables in an equation that can be used to predict profit or 
loss. 

Data gathering, representation, and building qualitative models. This refers to a 
set of related skills for collecting and examining information about values of 
properties selected during formulation. The intent of these activities is to 
understand empirical patterns of co-occurrence among variables of situations, 
leading students to an intuitive or qualitative model of the situation. For 
example, in the car wash situation, one might record tables of values and graphs 
showing how profit appears to covary with number of cars washed. The ability to 
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use multiple representations of relationships is key to successful comprehension 
of patterns in data (Kaput, 1989). 

• Translation of qualitative models into quantitative ones. Translation involves 
positing an equational representation that captures the observed patterns of 
interrelations of the formulated situational properties. For example, in the car 
wash situation, translation may involve postulating an equation that relates the 
variable for profit to variables representing the amount spent on supplies, the fee 
charged per car, and so on. This skill is related to hypothesis generation. 

* Inferencing or prediction involves answering questions about the situation at 
issue, either by manipulating equational representations or using less formal 
guess-and-test or extrapolation techniques that use graphs and tables. For 
example, given a problem in which one must infer the required duration of 
exercise to achieve a given weight loss, one can use an equational model for the 
situation or perhaps consult graphs and tables of values. This skill is related to 
hypothesis testing. 

PEDAGOGICAL GOALS 

Our pedagogical approach also introduced some departures from traditional algebra 
instruction. 



• Learning about mathematical tools in the context of natural situations. One 

general principle we adhered to was that students needed to learn algebra as a 
tool for solving problems in everyday life. Problems should be grounded in 
natural situations and problem solving should reflect the cooperative 
arrangements frequently found in the real world. High school algebra 
coursework often focuses on abstract principles (e.g., the commutative rule) and 
algorithms for solving abstract problems (e.g., the FOIL technique for multiplying 
factors). Word problems are often thinly disguised wrappers for equally abstract 
problems that seem to have little relevance to students’ concerns. The new 
NCTM math standards, other research (e.g., Fey et al., 1984; Schoenfeld, 1985; 
Collins, Brown, and Newman, 1989), and our own experience underscore the 
value of an approach grounded more in the real world. All coursework problems 
arose from familiar situations. We selected topics that were relevant to the lives 
of 1980s teenagers, such as after school jobs, rock concerts, renting limousines for 
the prom, and pollution of the local bay. 
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• Global before local skills. A second pedagogical principle underlying the 
curriculum was that students should be introduced to more global skills before 
learning the skills for local operations (e.g., Brown, Collins and Duguid, 1988). 
Thus, strategic goals for problem solving were introduced before specific symbol 
manipulation techniques or algorithms. For example, students were taught to 
specify the goals for each step in an equation solution before they were taught the 
mathematics for accomplishing the goal. As discussed below, the computer 
supported this approach by automatically doing the symbol manipulation. 

• Learning through inquiry as well as didactic presentation of material. Whenever 
possible, material was introduced in ways that fostered inductive instruction 
rather than by didactic presentation. The computer activity on covariation, for 
example, required that students input different values for several variables and 
then note the effect on an output variable. Students described how variables 
might affect each other (e.g., positively, inversely) on an accompanying 
worksheet. In this way, students were often able to use their strong informal 
skills for describing relationships to form intuitive mathematical models of 
relationships among variables before learning how to formalize them using 
equations and how to manipulate the equations. 

• Group collaborative learning. When appropriate, students worked cooperatively 
so that they could acquire skills in cooperative problem solving and take 
advantage of the opportunities for more effective discussion and exploration 
offered by this type of interaction (Slavin, 1986). Seatwork activities, such as an 
assignment to construct a qualitative model for a T-shirt sale and build a table to 
explore the effects of changes in the amounts of the variables, were often carried 
out in groups of four students. All computer work was conducted by pairs of 
students. 



THE COMPUTER TOOLS 

Different computer environments, including several common tools, were built to help 
students engage in each of the main activities involved in model building, formulation, data 
gathering and representation, and translation and inferencing using equations. More 
generally, these activities reflect the basic organization of the course materials and 
assignments. Early lessons and assignments focused on helping students determine which 
aspects of verbal situations represented the mathematical objects or variables of interest. 
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Then the students acquired skills for building input-output flow models of abstracted 
situations. Following this, students explored covariate relationships among variables by 
changing values of variables and observing the propagated effect on other variable values. 
Once students had an opportunity to develop a qualitative understanding of the relationships 
among variables, they were coached in expressing algebraic or quantitative models of the 
relationships. Finally, the students learned to manipulate formalized equations to answer a 
variety of specific questions about the situation. Here the embedded algebra ITS provided 
several levels of coadiing. Below, we discuss computer environments for each of these main 
activities in the overall lesson plan. 

Computer Environment for Formulation and Development of Qualitative Models 

Curriculum software was implemented in Franz Lisp on Sun Microsystems 
workstations (see McArthur et al., 1987; McArthur and Stasz, 1990, for a more complete 
description of the tutor architecture and functionality). For each assignment, students can 
select a situation from a menu of options (e.g., video store job, pizza party, weight estimation, 
boating). The tutor keeps track of which assignments and situations students have 
completed. The first time a student selects a situation he or she must always complete a 
model-building activity that specifies component variables. Subsequently, the computer will 
automatically rebuild the models, graphs, and tables that the student might have 
constructed in previous sessions. Figure 2 shows the model-building software interface for 
an early assignment where students are interested in determining the variables in a 
situation and gathering data to form a qualitative model of variable relationships. The 
situation to be modeled is described verbally in the “situation window” at the upper left. 
Notice that the situation description is not a verbal problem but an informal description of 
variable relationships. Later, the student will answer several specific questions based on 
this situation. Below the situation description is a “word graph” where the student inserts 
variables he or she has identified from a menu. The variables relate the dependent variable 
to various independent factors. Using the word graph, the student can input values for 
selected variables. The tutor then computes the appropriate values for other variables, based 
on an underlying equation that relates the variables. 

In the early assignment shown in Fig. 2, the student’s goal is to use guess-and-test 
strategies to understand co-occurrence relationships among variables and to answer specific 
questions. Students insert values into the word graph boxes and the environment provides 
several tools in which the data are automatically plotted to help the student gather and 
organize the data. These tools include tables of values (lower right) and cartesian graphs 
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(upper right). In addition, the student is guided in data gathering by specific problems 
(lower left). After students have a chance to freely explore varying values for variables, the 
problems generally constrain the student to explore covariation of pairs of variables. Here, 
students can vary the amount spent on “extras” (side dishes) while the pie cost, toppings cost, 
number of toppings, and discount coupon value stay the same. The data gathered in this 
“constrained exploration” permit the student to observe a positive linear relationship 
between the amount spent on the “extras” and the total bill. In addition to answering 
specific questions, these early assignments were to help students develop an intuitive or 
qualitative model of situations which in later sessions would be translated into a more 
quantitative, equational representation. 

Computer Environment for Translation and Answering Specific Questions 

In later assignments students could choose to work with the same situations they 
informally investigated on a previous day. As Fig. 3 shows, many computer tools (e.g., the 
situation description and word graph) are retained in later sessions. In addition, several 
tools appear to help students construct operators (*, /, +, -) and equations for the situation 
and to answer specific problems. Equations are first represented as “word equations” (below 
the word graph in Fig. 3), which look like word graphs with operators connecting the “boxes” 
that represent variables. Next, symbolic equations are computed from word equations by 
substituting letters for variable boxes (see Fig. 3). Subsequently, students are given a series 
of problems about the topic. The students can then use their equational models to generate 
answers. It is at this point that the equation-solving ITS comes into play, coaching the 
student through the manipulations required to make the desired inference. 

The algebra ITS embeds several kinds of knowledge that help it tutor students in 
solving equations. It knows the subject — the ITS is an expert in solving problems in simple 
symbolic algebra; it knows the student — it can infer limited diagnostic models of the 
students’ knowledge of algebra; and it knows something about teaching — it includes some 
simple rules of pedagogy that help it determine what information to provide and when to 
provide it. The ITS’s knowledge of algebra is articulate in a way that enables students to 
learn the different skills for solving equations in a systematic and top-down fashion. When 
solving an algebra problem, the ITS reasons separately about what goal to pursue (e.g., 
isolating the unknown), what operation implements the goal (e.g., adding 5 to both sides of 
the equation), and how to manipulate the symbols to accomplish an operation. Because the 
ITS algebra knowledge is articulate, we can structure specific student assignments to focus 
on each of these different levels of knowledge. In Fig. 3, for example, the student is solving 
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the symbolic equation developed to reflect the problem posed in the problem window at the 
lower left. Once developed, the symbolic equation automatically appears in the equation- 
solver windows on the right. In this example, the student is working at the goal level, 
specifying appropriate goals for each step of the equation. Goals include SIMPLIFYING the 
equation, ISOLATING the variable, COLLECTING terms, REDUCING and 
DISTRIBUTING. The ITS, in this example, is prompting the student for the goal for the 
next step in solving the equation Gower-right window) and, if the goal the student selects can 
be executed, the ITS will automatically implement the operation and manipulation necessary 
to carry out the goal. In this way, learning and problem solving involve role sharing between 
the computer and student. The current step in solving the equation is indicated by a box. 

Student-computer role profiles are defined for each assignment and determine what 
work will be done by student and by machine. This permits flexible cooperation. Consistent 
with our top-down pedagogical philosophy, in equation-solving assignments the standard role 
profile dictates that the student will first select only goals and that the ITS will supply 
operations and manipulations. Later, students become responsible for operations, and in the 
final assignments students do goals and manipulation. Thus, only at the end of the course 
will students do the “symbol crunching” that is central to most introductory algebra courses. 

The algebra ITS also embeds several other pedagogical rules. In the model-building 
activities, students can select correct variables, operators, or symbols. If they make two 
errors in a row, the computer supplies the correct term. In the data collection/covariation 
activities, students continue collecting data until they feel they have the right answer. The 
activity is open-ended, allowing for unlimited exploration, and no coaching is provided. In 
the equation-solving environment, students have access to the equation solver by asking for 
“help.” The tutor will specify the conditions for that step in the equation and the goal, 
operation, or symbol manipulation as appropriate for the problem. If further help is 
requested, the tutor will illustrate how to perform the step. Students can also ask the tutor 
to check whether any equation in the solution tree is correct. 

TEACHER AND STUDENT MATERIALS 
Teacher Materials 

To provide support for teachers and insure that the course material would be taught as 
intended, we developed specific lesson plans (see Appendix A for full set of curriculum 
materials). Introductory material in the curriculum package describes the course goals and 
the pedagogical techniques to be used. A typical lesson begins with a review of a previous 
topic. The teacher next introduces the topic of the day and provides practice exercises for the 
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topic that are completed by the class and then reviewed with the teacher. Then individual 
(or small group) practice is conducted either through noncomputer or computer assignments. 
Each lesson plan contains the goals for the lesson and an almost scripted set of activities to 
carry out, including board examples for the teacher to use. Specific lessons are included for 
introducing the various computer environments. Teachers also have available printed review 
sheets — used for warm-ups and practice activities, noncomputer assignment sheets, 
computer assignment sheets, and vu graphs of materials for whole class review. The lesson 
plans, computer assignments, and tests for each of the two units were developed by the 
research staff with teacher input, and most of the review assignments and noncomputer 
assignments were developed by the teachers with staff input. We hoped by this process to 
thoroughly acquaint the teachers with the course goals, pedagogy, and materials. 

Student Materials 

Twenty assignments were prepared for students— eight computer assignments and 12 
noncomputer assignments. Students received booklets that contained all of their assignment 
worksheets. Each computer assignment was accompanied by a brief worksheet on which 
students could record the situations selected and the results of their explorations and note 
the methods they had used. These worksheets were designed primarily to provide hardcopy 
records of student computer work for teacher use and to make computer and noncomputer 
assignments more alike. Whenever possible, noncomputer assignments parallel computer 
assignments but do not necessarily share the same features. For example, in exploring 
covariation on the computer, an underlying equation permits the tutor to compute a missing 
value. In paper assignments, the equation is explicit and is computed by students. In 
equation solving with the tutor, the student can solve an equation by specifying goals while 
the computer handles all the symbol manipulation. In noncomputer assignments, students 
can specify a goal for a specific solution step, but until they can perform the symbol 
manipulation corresponding to the goal, they cannot move the solution ahead to the next 
step. 

Noncomputer assignments provide practice in (i) specifying goals and operations for a 
given step in a solution; (ii) completing the symbol manipulation for the specified goal/or 
operation; (iii) solving an entire problem by specifying the goals and completing the symbol 
manipulations; and (iv) symbol manipulation (without goal specification). Generally, the 
opening problems in a noncomputer assignment are situation-based (like the computer 
problems), whereas the remainder are more abstract and provide students computational 
practice. 
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A TYPICAL LESSON 

To see how all the different pieces of the curriculum package fit together, imagine, for 
a moment, a typical classroom day during the second week of the curriculum. Students 
arrive and find the day’s activities listed on the board: Warm-up; Introduction to graphing; 
Worksheet #4 or Computer Assignment #3. With a bit of noise, some com chips snatched out 
of a bag, and a search for pencils, the class settles down to do the warm-up review exercise 
while the teacher checks attendance. A word graph on the board represents the profit to be 
made from a Black Student Union/Mexican Association dance: 

# of tickets ticket price expenses ==> profit 



Below the graph is a table in which price and expenses are held constant. Students are in- 
structed to complete the table, calculating the profit based on different numbers of tickets sold: 



# of tickets 


* ticket price 


- expenses 


= 


profit 


40 


$3.00 


$75.00 




$45.00 


50 


3.00 


75.00 






75 


3.00 


75.00 






100 


3.00 


75.00 






150 


3.00 


75.00 






200 


3.00 


75.00 







The teacher asks questions about the table to illustrate basic concepts: “As you increase the 
number of tickets, what happens to the profit?” (covariation); and, “As the number of tickets 
increases what happens to expenses?” (effects when one variable is held constant). Moving 
on to the topic of the day, the teacher draws a graph relating profit to tickets and with 
student help plots points from the table: 
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The teacher asks how much money could be made if 250 tickets were sold and how 
many tickets would have to be sold for a profit of $675.00. Students answer the Questions 
and the teacher asks them to explain how they arrived at the answer, thus, forcing them to 
articulate their reasoning processes. 

The teacher uses the noncomputer assignment sheet to continue the explanation. On 
an overhead projector, she displays the first section of this worksheet, which contains a 
situation discussing blood pressure as a function of age, a table showing several age inputs 
and resultant average blood pressures, and a graph plotting the age inputs against blood 
pressure (see the Student Materials, Assignment M4, p. 1, in Appendix A). The assignment 
continues with questions aimed to guide students through reading the table and the graph. 
The teacher goes through a few questions with the whole group to check their understanding. 
The assignment contains an additional situation (with relevant tables, graphs, and 
questions) designed to help students learn to use tables and graphs to solve problems. 

For the second half of the period, half of the students complete work on the 
noncomputer assignment and the other half work in preassigned pairs on the computer 
assignment (the following day, students will switch assignments). At the computers, 
students login and select a situation to work on. Anna and Sara select Pizza Heaven, a 
situation they enjoyed working on last time. Since they have worked on this situation before, 
the computer displays the word graph they made previously and then the first problem of 
today’s assignment: varying the cost of extra side dishes to achieve a total bill of $15.00 (see 
Fig. 2, which shows how values are input into word graphs). The girls hesitate for a moment 
and wonder what to do first. The teacher, circulating through the room, notices their puzzled 
looks, and reminds them to look at the instruction window. Working as a team, Anna clicks 
the mouse on each word graph variable box, and Sara types in the amount given in the 
problem for each variable — the cost of the pizza pie, the number of toppings, the cost per 
topping, and the amount of a discount coupon. The girls discuss the situation and decide 
that all they want extra is drinks, so they input $2.00 for “extras.” The computer computes 
the total bill, and the girls find that they are still $4.00 under the $15.00 total. All of the 
data input into the word graph is recorded in a table, and a graph of the amount input for 
“extras” against the bill total is also displayed. The girls continue inputting different 
amounts for “extras” until they achieve the $15.00 total bill. The computer then directs them 
to complete the first part of their worksheet for this assignment, M3 (see Student Materials, 
Assignment M3, part A, in Appendix A). The worksheet asks students to record the problem 
and their answer. Then it asks how they arrived at the correct answer and to explain to 
someone else a technique for getting the answer. 
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When Anna and Sara complete their worksheet, they continue with the next computer 
problem, which requires them to use a graph to determine the price of pizza for a given bill 
value if all other variables are held constant. The girls click on the graph’s “bill” axis at the 
location that corresponds to the value for the bill, and the computer visually computes the 
corresponding value on the “pizza” axis. The computer then asks them to record their 
exploration on a worksheet, which also asks what “pizza” input would produce a very high 
“bill” output. Since there is still ten minutes left in the period, the girls select a second 
situation — “Heal-the-Bay” — and complete similar problems. When they are finished, they 
logout and return to their seats. 
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3. IMPLEMENTATION 



We implemented the novel curriculum over five weeks in the fall of 1988 in a local 
high school. In this section, we describe the teachers and students who participated in the 
study, teacher and student training, and classroom operations and data-gathering 
procedures used in this study. 

THE STUDY PARTICIPANTS 
Teachers 

The curriculum module was taught by two teachers who serve as consultants to the 
RAND Algebra Tutor projects and were members of the curriculum development team. 
Teacher A is a young woman with several years of secondary school math instruction 
experience. She is interested in curricular innovation and the use of technology. She is a 
member of the UCLA California Mathematics Framework Project, helped launch a computer 
program at a Los Angeles Unified School District magnet school, and was designated by the 
high school to design an innovative “Math A” course curriculum. As a member of the 
project’s curriculum development team, she had responsibility for developing many of the 
noncomputer assignments and extensively reviewed the teacher lesson plans and software 
specifications. 

Teacher B is a senior member of the same high school faculty and has over 20 years of 
mathematics teaching experience. She has been actively involved in curricular reform and 
designed a statistics course for the high school. She has made presentations at national 
conferences on such topics as "Liberal Arts Math,” "Integrating Computers in Math Courses,” 
“Finite Math Topics,” and "Statistics and Probability in the Traditional Course of Study.” In 
recognition of her expertise, she was selected as a Woodrow Wilson Fellow in Statistics at 
Princeton and received her school district’s outstanding faculty award. Teacher B’s classes 
participated in both of the Algebra Tutor Project’s previous field studies (see McArthur and 
Stasz, 1990; Robyn et al., 1989; Stasz et al., 1989). During the curriculum development 
phase of this study, Teacher B helped develop and review materials but took a less active role 
in the project than Teacher A. 

Students 

Participating students were enrolled in either Math A or regular algebra classes. 

Math A is the first year of a new two-year course implementing the recent California Math 
Framework (California State Department of Education, 1985). It features several discrete 
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units — probability, symmetry, statistics, algebra, and geometry. The course is intended to 
expose students to enough algebra and geometry to enable them to continue in a college- 
focused academic plan. The course targets less-successful students who typically are low 
achievers in an algebra class. Our curriculum module comprised the introductory algebra 
unit in the Math A course. 

All four Math A classes participated in the study. Two sections were taught by each of 
the teachers. Parent consent was obtained for participation in the data-collection activities. 
The average class size was about 32, although actual daily attendance was about 28 
students. To provide comparative information, four regular algebra classes were asked to 
participate in the data-collection activities. Classes were selected to match as closely as 
possible the periods of the Math A classes (one class time could not be matched), and each 
class was taught by a different teacher to avoid a teacher effect. All of these classes 
progressed at about the same pace through the same textbook. At the time of the study, the 
algebra students were working on linear equations and factoring. 

TEACHER AND STUDENT TRAINING 

Since both pilot teachers were members of the curriculum development team, formal 
teacher training was required only for the computer component of the course. The software 
was available for teacher review as it was developed. During two laboratory sessions, 
materials designed to train students to use the tutor for the modeling and equation 
construction environments were demonstrated by staff members. The teachers then coached 
students through the relevant computer assignments. The equation-solving software was not 
available for laboratory piloting but was available to the teachers shortly before it was used 
in the classroom. 

The two teachers differed dramatically in the amount of time they spent learning 
about the computer software and hardware. The difference seemed to be a function of the 
teachers’ own interests, the constraints of teaching and class preparation duties, and other 
professional responsibilities. Teacher B spent approximately five hours in computer training 
(the time spent at the two lab sessions) and Teacher A spent about ten hours (including 
additional lab sessions with students and individual exposure), and about 20 additional 
hours (approximately an hour during each day of the implementation) debugging software. 

The teachers received little preliminary training on troubleshooting computer 
problems; project staff attended every class and were expected to handle any emergencies. 
Just before the curriculum module began, Teacher B decided that she did not want to be 
responsible for student work at the computer, so project staff took over this task. The 
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teacher felt too pressured to monitor the computer work in addition to teaching new course 
material. Teacher A, on the other hand, decided to handle all class activities with a minimal 
amount of help from project staff. Thus, she received additional instruction about 
troubleshooting computer problems. 

Student training materials enabled the teacher to instruct the whole class at once. 
Using an overhead projector and transparencies of the Sun workstation operation, the 
teacher walked the students through login procedures and the different computer 
environments and activities before the students actually sat down at the computers (see 
Teacher Lesson Plans, pp. 7-9 and pp. 24-25, in Appendix A). Teacher A spent about an 
hour introducing the computer and various environments and Teacher B spent about 20 
minutes. 

CLASSROOM OPERATIONS AND DATA-GATHERING PROCEDURES 

Seven Sun 3/50 workstations were installed in the Math A classroom. Project staff 
were present in every class to handle any computer operational problems that arose. The 
curriculum was designed to span 13 days, but teachers were not required to adhere to a rigid 
schedule. Teachers were encouraged to set their own pace and they taught the course in 20 
days. Class observation data indicate that teachers spent about ten days on the modeling 
portion of the curriculum and about 15 days on equation solving; this time was 
proportionately equal to the original schedule. 

We used multiple methods to assess the curriculum, including mathematics and 
algebra achievement tests, two unit tests, background and attitude questionnaires, classroom 
observation instruments, and computer records. (See McArthur and Stasz, 1990, and Stasz 
et al., 1989, for further description of our overall approach to data collection). These 
instruments are included as Appendix B. Students took an arithmetic skills test to measure 
arithmetic ability and a pretest and posttest that assessed student-experience with and 
attitudes toward computers and mathematics and their problem-solving skills. The skills 
test included traditional abstract equation-solving problems, as well as novel problems 
featured in the curriculum (e.g., identifying variables in a natural situation, constructing an 
equation to represent a natural situation, solving problems related to the situation, and 
specifying problem-solving strategies). The set of background questions (e.g., gender, age, 
grade in school, past mathematics courses, feelings about algebra, computer experience) have 
been tested and refined over the course of several Algebra Tutor projects. The attitude 
questions, also used in our previous research, were developed by the research team or 
adopted from other studies (e.g., Schoenfeld, 1985; Carpenter et al., 1980). These 
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instruments were also given to four Algebra 1 classes. The pretest was administered just 
after the algebra students completed lessons on linear equations. The posttest, given five 
weeks later, followed lessons on factoring. Thus, algebra students’ knowledge of equation 
solving at pretest was similar to the Math A students’ knowledge at posttest. 

Math A students also completed two unit tests — one on identification of variables and 
covariation and the other on equation solving. These are described more fully in Sec. 4. The 
posttest for Math A students also included student ratings of the course. Computer records 
automatically recorded student time on the computers and assignments completed. To 
assess implementation of the novel curriculum, project staff kept daily class observation 
records of the topics presented, the effectiveness of the teacher’s presentation, student 
experiences with the class and computer activities, and general computer operation. These 
records also included teacher comments about the course and general student progress. 
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4. EVALUATION OF THE CURRICULUM 



In this section we discuss our overall approach to evaluating the curriculum and 
present student outcomes. We examine these outcomes with respect to intended and actual 
implementation of the curriculum, as indicated in classroom observations of teaching and 
learning. Finally, we examine students’ evaluation of the course and assess the curriculum’s 
limitations. 

EVALUATION DESIGN 

The evaluator seeking to determine the effectiveness of any instructional program has 
a wide range of alternative designs from which to choose (e.g., Campbell and Stanley, 1966; 
Cook and Campbell, 1979). Choosing which is best depends on a number of factors, including 
the types of decisions (and decisionmakers) on which the evaluation focuses and the 
feasibility of implementing the design (Shavelson et al., 1986). 

Generally, two questions are raised in most curriculum evaluations: How much do 
students gain in knowledge, skills, and attitudes from the instructional intervention? Is 
what they learn the same as what students learn in traditional courses that are intended to 
teach the same subject matter (Shavelson et al., 1986)? In this study, we weight the question 
of knowledge gain as more important than the question of equivalence of outcomes. We 
believe this focus appropriate for a curriculum development project which is innovative in 
content, pedagogical approach, and supporting instructional materials. Thus, our evaluation 
approach is more formative than summative (see Flagg, 1990, for approaches to formative 
evaluation of educational technologies). The aim is to pilot test a prototype curriculum and 
gather data to understand which aspects of the curriculum were successful and which need 
revision. 

A third evaluation question, less often addressed, concerns implementation: What 
factors in the implementation process influence the program’s success or failure? This 
question is particularly important for instructional programs that include new technologies, 
since the technology itself may pose unique barriers to implementation in a school setting 
(e.g., Newman, 1989; Robyn et al., 1989). Thus, our evaluation was sensitive to teachers’ and 
students’ attitudes about and use of the computer-aided aspects of the curriculum as well as 
their overall evaluation of the instructional activities. 

Although the question of comparability of student outcomes between Math A and 
regular algebra students was of secondary importance, our design included regular algebra 
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students as a “comparison” group. 1 We were interested in how the Math A students 
compared with “regular” algebra students on several dimensions, such as arithmetic 
achievement, algebra knowledge, demographics, attitudes, and other background 
characteristics. Knowledge of these types of differences might enable us to better understand 
effects (or lack of them) from the novel curriculum. Several of our attitude questions assess 
attitudes and beliefs about mathematics that reflect student experiences with traditional 
mathematics instruction; for example, that mathematics is mostly memorizing. If Math A 
students’ attitudes show greater changes relative to algebra students in beliefs of this type, 
then we have some indication that the novel curriculum may have helped effect such 
changes. 

STUDENT CHARACTERISTICS AND ATTITUDES 

All 110 students in the four Math A classes consented to participate in the study. In 
the comparison group, 105 algebra students participated and three declined. Table 1 shows 
how these students compare with respect to age, grade level, gender, and pre-course 
arithmetic ability. 

Math A students are older (F = 4.86, p < .03) and, on average, are in higher grade 
levels than algebra students (F = 4.65, p < .03). They also have significantly weaker 
arithmetic skills than algebra students (F = 71.52, p < .001). Algebra classes had a higher 
proportion of male students than the Math A classes. About 64 percent of the algebra 
students, compared to about 41 percent of the Math A students, expected a grade of B or 



Table 1 

Student Characteristics 





Mean 


Mean 


Percent 


Arithmetic 




Age 


Grade 


Male 


Ability 


Math A 


15.1 


9.8 


52% 


5.0 


Algebra 1 


14.8 


9.6 


57% 


7.5 



Total possible = 10. 



! We use the term “comparison” group rather than “control” group because we are not using the 
data to examine equivalence of outcomes in the strict sense. We also acknowledge several differences 
between these groups of students that temper any comparisons made. Students were not randomly 
assigned to classes, and Math A was targeted at less-successful students who would typically be low 
achievers in a regular algebra class. According to the teachers' reports, confusion in the counseling 
office resulted in students of very low ability (including “learning disabled” students) being scheduled 
for Math A along with a few students who had already had algebra or geometry. Thus, Math A and 
algebra students constituted different student populations. In addition, the classes had very different 
instructional goals and curricula, although both courses covered some of the same material (e.g., 
solving linear equations). 
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higher in the course. Algebra students liked algebra more than Math A students (71.4 and 
51.8 percent, respectively; chi square 16.75, p < .001). Twenty percent of the algebra 
students judged algebra to be "easy,” compared to only 4 percent of the Math A students (chi 
square = 10.75, p < .005). 

We asked students several questions about their feelings toward mathematics and 
toward computers. Students answered on a four-point scale (1 = strongly disagree to 4 = 
strongly agree). Table 2 shows percentage agreement (percentage responding “3” or “4”) with 
these statements for Math A and algebra students. Table 2 indicates that algebra students 
may have more confidence in their mathematics ability, although the difference between 
Math A and algebra students is not significant (item 1). Similarly, both groups look for 
external verification of their work (item 3), but the Math A students appear more disposed to 
do so. A majority of students in both groups think computers can improve algebra 
instruction (items 2 and 8). Items 4 and 10 assess how highly students value mathematics. 
Responses here appear somewhat inconsistent. About a quarter of the students in both 
groups saw little value in mathematics (item 4), but over half thought algebra was useful in 
life (item 10). Algebra students saw significantly more value in learning algebra than did 
Math A students. A second attitude about mathematics learning concerns discovery versus 
memorization (item 11). We would like students to see mathematics as a logical structure 
rather than as just a body of disconnected "facts.” Most students believe that mathematics is 
mostly memorizing. In spite of this, a high percentage of students also believe that discovery 
is possible (item 6). Items 5 and 9 concern similar attitudes about mathematical thinking. 
Clearly, a majority of students believe that trial and error is a useful strategy. Few students, 
on the other hand, had faith in their ability to figure out a problem. This is perhaps 
consistent with their view that mathematics is mostly memorizing. If students believe that 
mathematics consists of memorizing formulas, then they will not go back to "first principles” 
to reason the answer to a question. Finally, item 12 concerns students’ attitudes toward 
mathematics as a body of knowledge. Math A students appear significantly more likely to 
view mathematics as comprising unrelated topics than do algebra students. 

We also asked students about their experiences with computers (see Table 3). Overall, 
students have a wide variety of experience with computers. Algebra students are 
significantly more likely to use computers outside of school (chi square = 13.08, p < .001) and 
to use computers to do their homework (chi square = 3.88, p < .05). A large proportion of 
students have taken courses about computers and nearly 40 percent of students in both 
groups have written computer programs. 
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Table 2 

Attitudes Towards Math at Pretest 



Percent Agreement 


Item 


Algebra 


Math A 


1. I am good at math 


72.4 


64.2 


2. Many algebra courses could be 
improved with computers 


72.4 


69.4 


3. In math, to know if you have the 
right answer, you must check with 
the teacher, a computer, or the 
book 


53.3 


64.2 


4. I can get along well in everyday 
life without mathematics 


22.9 


23.1 


5. A problem is easy to figure out 
even if IVe forgotten exactly how to 
do it 


28.6 


25.7 


6. In mathematics you can be 
creative and discover things for 
yourself 


60.0 


64.1 


7. There is one right way to solve 
most algebra problems 


36.2 


33.3 


8. Using computers to teach algebra 
is a bad idea 


17.1 


20.0 


9. Trial and error can often be used 
to solve an algebra problem 


58.2 


68.0 


10. It's important to learn algebra 
because algebra knowledge is 
useful in everyday life 


67.3 


54.1* 


11. Learning mathematics is mostly 
memorizing 


69.5 


68.8 


12. Mathematics is made up of 
unrelated topics 


23.3 


36.8** 



NOTE: Because of missing data. Ns range from 103-105 for algebra 
students and from 96-110 for Math A students. (* = chi square 3.87, p < .05; 
** = chi square 4.52, p < .03.) 



In summary, the Math A students differ somewhat from traditional algebra students. 
They are older, in higher grades, and less skilled in arithmetic than algebra students. Math 
A students also like algebra less and are less likely to judge it “easy.” Students in both 
groups have similar attitudes about mathematics, but algebra students see more relevance 
for algebra knowledge in everyday life. Algebra students have more experience with 
computers outside of school than Math A students. 
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Table 3 

Computer Experience 





Percent Responding "Yes” 


Have You Ever. . . 


Algebra 
(N = 105) 


Math A 
(N = 110) 


Taken a course about computers 
or computer programming? 


70 


66 


Used computers outside of school 
other than for playing games? 


76 


61 


Used computers in other classes? 


60 


49 


Used a computer to do your 
homework? 


36 


24 


Written a computer program? 


42 


39 



STUDENT LEARNING 

To assess changes in student learning, we administered an algebra achievement test 
to the Math A students before and after the curriculum unit, and, for purposes of comparison, 
to the algebra students after their instruction in solving simple linear equations. 2 Students 
also took an arithmetic test to determine initial arithmetic ability, apart from algebra 
knowledge. Finally, Math A students took two “unit” tests that measured learning of 
concepts specific to the new curriculum. Unit test results will be discussed in detail below. 

Table 4 shows test scores for the 65 Math A students and 64 algebra students who 
completed all of their respective tests. This comparison eliminates students who entered the 
courses late, dropped out of the courses, or were absent on testing days. 3 

Algebra students scored significantly higher than Math A students on the arithmetic 
test (F = 33.37, p < .001) and algebra posttest (F = 24.09, p < .001). What is most striking, 



2 We gave two tests to the Algebra I classes as well as the Math A classes. However, the algebra 
students cover the material on linear equations quite early in the semester, so the first of the tests was 
in effect a posttest, occurring immediately after students had received instruction in the relevant topics. 
The results of the second test, which was given at the end of the semester, are quite similar to the first 
and show a slight, statistically insignificant decrease toward the end of the semester (change score = -.2 
on a test with 12 possible points). Thus, for purposes of comparing achievement, we use the first test 
given to the algebra students and the second given to the Math A students. On the other hand, where 
changes in attitude are concerned, we felt justified in using both algebra students' tests, since the 
issues in question relate to the way in which algebra is taught generally and is not content-specific. 

3 We conducted separate analyses to determine if these scores were significantly different for the 
group of 44 algebra and 49 Math A students who missed one or more of these tests. T-tests between 
total scores of students taking all tests as compared with those missing, for each of the three tests, were 
not significant. Thus, there is no bias with respect to achievement between these two groups. 
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Table 4 

Arithmetic and Algebra Achievement Scores for All Students 
(Means and Standard Deviations) 





Arithmetic 


Pretest 


Posttest 


Change 

Score 


Students 


Mean 


S.D. 


Mean S.D. 


Mean 


S.D. 


Mean SD. 


Math A 


5.17 


2.38 


1.80 1.72 


4.75 


2.86 


2.94 2.18 


Algebra 


7.42 


2.04 




7.07 


2.50 





NOTE: Maximum possible scores are 10 on the arithmetic test and 12 on the pretests or posttests. Change 
score is the posttest minus the pretest. 



however, is that both algebra students and Math A students did so poorly on the posttest, 
correctly answering only about half of the questions on average. Nevertheless, we were 
encouraged to see that the Math A students did show significant improvement from pretest 
to posttest (t = 10.86, p < .001). Moreover, among the Math A classes, Teacher A’s students 
did significantly better than Teacher B’s (change score 3.65 and 2.17, respectively; t = 2.88, 

p < .01). 

We also examined group differences on two subsets of items: Six “innovative” items 
and six “traditional” items. Innovative items, for example, assessed students’ ability to select 
(from a list of operations) the operation that would transform the equation to the next 
appropriate step or to identify the variables in a word problem. Traditional items required 
students to solve an equation presented in symbolic form (e.g., 3(Y — 2) = 12; 5.2T - 10.3 = 
10.5). Algebra students completed about three (of six) traditional items correctly on the 
posttest. Math A students, on the other hand, do slightly better than algebra students on the 
innovative items at posttest, and show small gains from pretest to posttest. 

For Math A students, the small gain in innovative items is encouraging, since the 
novel curriculum stressed these skills over solving symbolic equations. Analysis of subset 
test scores by teacher indicates that Teacher A’s students had significantly higher change 
scores than did Teacher B’s on both types of items. 

Given such low test scores, these results are not very compelling from a curriculum 
developer’s perspective. However, we have other data on the Math A students — teacher 
evaluations, unit tests, class notes, tutor usage — to determine which aspects of the 
curriculum were more successful than others and what kinds of changes might be made. We 
examin e these data in more detail below. 
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MATH A STUDENT OUTCOMES AND CHANGES 

Since the primary purpose of this evaluation is to assess how the novel curriculum 
changed students’ knowledge, skills, and attitudes, below we analyze data for the Math A 
students alone. First, we more closely examine their achievement on the algebra test and 
unit tests. Then we assess changes in student attitudes. For these analyses, we are also 
interested in whether any teacher differences arise. 

Math A Achievement 

Although student performance on the algebra achievement test was low overall, some 
classes did better than others. Table 5 shows algebra achievement data by teacher. Both 
teachers’ students had had equivalent arithmetic and algebra knowledge at pretest. Teacher 
A’s students did significantly better at posttest than Teacher B’s students (t = 1.99, p < .05). 
Change scores (posttest minus pretest) for the total test and various subsets of item types 
indicate that Teacher A’s students outperformed Teacher B’s students across the board. 

Unit Test Results 

Two unit tests were administered during the course. The first test was given after 
students completed curriculum topics on identifying variables, developing qualitative models, 
and exploring covariation using tables and graphs. The second unit test was given after 
completion of the full curriculum module and assessed the construction of quantitative 
models (equations), identification of target variables, equation solving (at goals, operations, 
and symbol manipulation levels), problem checking, and interpretation of answers. 

Appendix B contains both unit evaluations. Both unit evaluations were paper-and-pencil 
tests that mirrored noncomputer and computer assignments. 



Table 5 

Algebra Achievement by Teacher 





Teacher A 


Teacher B 


Significance 




Mean 


SJ). 


Mean 


SJ>. 


Level 


Arithmetic 


4.82 


2.14 


5.25 


2.60 


n.s. 


Pretest 


1.63 


1.47 


1.72 


1.98 


n.s. 


Posttest 


5.33 


3.10 


4.15 


2.75 


p <.05 


Total change 


3.65 


2.26 


2.13 


2.29 


p <.01 


Equation-solving 


1.76 


1.65 


0.92 


1.66 


P <.02 


change 

Innovative change 


1.89 


1.30 


1.21 


1.32 


p <.02 



NOTE: Maximum possible scores are 10 on the arithmetic test, 12 on the pretest and 
posttest. Change, scores represent average change from a base of six items each. 
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Table 6 shows average student scores, by teacher, on each of the two tests. In contrast 
to their performance on the algebra achievement test, both teachers’ students did fairly well 
on the first unit tests, although less well on the second. Teacher A’s students scored slightly 
higher than Teacher B’s students, but this difference is not statistically significant. 

To examine student achievement in greater detail, we constructed several subsets of 
items and, for the sake of comparison, standardized subset scores on a ten-point scale (see 
Table 7). These data reveal several important differences. First, students were twice as good 
at solving symbolic equations that were situated in a real-world problem than solving similar 
equations presented abstractly. In a “situated” equation (see Fig. 3 for an example), the 
student is asked to solve a symbolic equation derived from the word problem and word graph. 
The student is also asked to identify the target variable and the goal before solving the 
problem. Students did better on these types of problems than on similar equations (e.g., 3Y + 
9 = 12) presented alone. This finding reinforces the belief that problem-solving should be 
grounded in meaningful contexts so that students can learn to “mathemetize” or construct 
the links between formal algebraic expressions and the actual situations to which they refer 
(e.g., Clement, 1982). It appears that meaningful problems, such as those included in our 



Table 6 

Unit Test Results by Teacher 







Teacher A 






Teacher B 






N 


Mean 


S.D. 


N 


Mean 


S.D. 


Unit Test #1 


49 


5.01 


1.39 


58 


4.80 


1.46 


Unit Test #2 


55 


4.20 


2.23 


53 


3.72 


2.61 


NOTE: Maximum score for #1 and #2 were 6 and 11 points, respectively. 



Table 7 

Standardized Subtest Scores by Teacher 


Subtest 


Teacher A 


Teacher B 




Mean 


SD. 


Mean 


SJ>. 


Symbolic equations 


2.27 


3.05 


2.59 


3.46 




Situated equations 


4.80 


3.14 


4.11 


3.56 




Find target 


7.63 


3.71 


5.75 


3.97 


p < .01 


Choose goal 


5.89 


3.18 


4.60 


3.48 


p < .05 


Choose operator 


3.45 


3.95 


2.36 


3.74 




Check answer 


1.64 


2.73 


1.51 


2.99 




Interpret answer 


3.67 


3.98 


3.58 


3.62 





NOTE: Maximum score on each subset is 10. 
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curriculum, can improve over current mathematics instruction, which does not engage 
students’ interpretive and meaning construction capacities (cf. Resnick, 1987). The 
challenge, then, is to determine what constitutes a “meaningful” problem to teenage 
mathematics students. We obtained some feedback from students on this and other aspects 
of the curriculum, which we report below. 

Table 7 also indicates that students did fairly well at identifying the target variable 
and choosing the goal for a particular step. Furthermore, Teacher A’s students did 
significantly better than Teacher B’s on these two types of items. Our classroom observation 
data indicate that Teacher B did not reinforce target identification in her presentation, 
whereas Teacher A covered this topic in at least two different sessions and received an 
average rating of 2.5. 4 

Students did less well at checking and interpreting answers. The low score for 
“checking” is somewhat disappointing. There is some evidence that successful math learners 
engage in more metacognitive behaviors, such as checking. It seems reasonable that a less- 
routinized approach to mathematics, such as that fostered in our curriculum, could 
substantially improve learning, including developing metacognitive or strategic behaviors for 
solving problems (cf. Resnick, 1987). However, it appears that students failed to check even 
when they were specifically instructed to do so. It may be that checking is not a typical 
activity for these students and that the instruction alone did not induce them to pick up this 
metacognitive habit. Our observational data indicate that neither teacher reinforced 
checking during their classroom instruction: Only one observer noted any presentation of 
this topic, to which a rating of “1” (not effective) was assigned. 

Changes in Attitudes 

Next we examine changes in attitudes toward mathematics. Here, we are interested 
in whether Math A students’ attitudes change in the anticipated direction (either an increase 
or decrease in agreement, depending on the statement) relative to algebra students. This 
analysis compared what percentage of Math A students’ (N = 65) attitudes changed relative 
to the algebra students (N = 64). 

4 Observers in each class noted topic numbers from a list of possible topics and rated the 
teacher's presentation on a three-point scale: 1 = not effective, 2 = satisfactory, and 3 = very effective. 
Observers were members of the research team who had helped develop the curriculum and thus were 
knowledgeable about curriculum objectives and intent. Although their ratings may be very subjective, 
they do give some indication of whether the teacher covered the curriculum material or taught it as 
originally intended. 
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